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Abstract 

This paper deals with the long run average continuous control problem of piecewise deter- 
ministic Markov processes (PDMP's) taking values in a general Borel space and with compact 
action space depending on the state variable. The control variable acts on the jump rate and 
transition measure of the PDMP, and the running and boundary costs are assumed to be positive 
but not necessarily bounded. Our first main result is to obtain an optimality equation for the 
long run average cost in terms of a discrete-time optimality equation related to the embedded 
Markov chain given by the post-jump location of the PDMP. Our second main result guaran- 
tees the existence of a feedback measurable selector for the discrete-time optimality equation 
by establishing a connection between this equation and an integro-differential equation. Our 
final main result is to obtain some sufficient conditions for the existence of a solution for a 
discrete-time optimality inequality and an ordinary optimal feedback control for the long run 
average cost using the so-called vanishing discount approach (see [lf|, page 83). 
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1 Introduction 



A general family of non-diffusion stochastic models suitable for formulating many optimization 
problems in several areas of operations research, namely piecewise-deterministic Markov processes 
(PDMP's), was introduced in [6|, and [8]. These processes are determined by three local charac- 
teristics; the flow <j>, the jump rate A and the transition measure Q. Starting from x the motion of 
the process follows the flow <j)(x, t) until the first jump time T\ which occurs either spontaneously 
in a Poisson-like fashion with rate A or when the flow <j)(x, t) hits the boundary of the state-space. 
In either case the location of the process at the jump time T\ is selected by the transition measure 
Q((p(x,Ti), .) and the motion restarts from this new point as before. A suitable choice of the state 
space and the local characteristics </>, A, and Q provide stochastic models covering a great number 
of problems of operations research [8] . 

Closely related to the PDMP's are the so called Markov decision drift processes, introduced by 



Hordijk and Van der Duyn Shouten, see 18|], 19(], 20|]. Their approach is to analyze the control 



problems for this class of processes via time-discretizations and deterministic grid approximations 
of the original process, combined with the theory of Markov Decision Processes. Yushkevich [261.1271] 
considers similar control problems but adopts a variational approach which yields to some integro 
differential Bellman optimality inequations and characterization of the value function based on the 
concept of absolute continuity. By considering some stronger continuity assumptions and under 
the framework of the PDMP's, Dempster and Yej^, QlJ] provide some characterization results for 
the value function based on a generalized Bellmam equation which involves the Clark generalized 
gradient. There exist two types of control for PDMP's: continuous control and impulse control. 
This terminology has been introduced by M.H.A. Davis in [£], page 134] where continuous control is 
used to describe situations in which the control variable acts at all times on the process through the 
characteristics (cj), A, Q) by influencing the deterministic motion and the probability of the jumps 
by opposition to impulse control that intervenes on the process by moving it to a new point of the 
state space at some times specified by the controller. 

This paper deals with the long run average continuous control problem of PDMP's taking values 
in a general Borel space. At each point x of the state space a control variable is chosen from a 
compact action set \J(x) and is applied on the jump parameter A and transition measure Q. The 
goal is to minimize the long run average cost, which is composed of a running cost and a boundary 
cost (which is added each time the PDMP touches the boundary). Both costs are assumed to be 
positive but not necessarily bounded. As far as the authors are aware of, this is the first time that 
this kind of problem is considered in the literature. Indeed, results are available for the long run 



average cost problem but for impulse control see Costa [4J, Gatarek 14| and the book by M.H.A. 
Davis 0] (see the references therein). On the other hand, the continuous control problem has been 
studied only for discounted costs by A. Almudevar [H, M.H.A. Davis 0,0], M.A.H. Dempster and 



J.J. Ye @,E3l, Forwick, Schal, and Schmitz [O}, M. Schal 0], A. A. Yushkevich 0,3 . The reader 



may consult the book by M.H.A. Davis [8j] and especially the sections at the end of the chapters 4 
and 5 for a complete survey on stochastic control problems for PDMP including theoretical results 
and applications. 

Our approach to study the long run average continuous control problem of PDMP's is to follow 
the idea by M.H.A. Davis and reformulate the optimal control problem of the PDMP as an equiva- 
lent discrete-time Markov decision model in which the stages are the jump times T n . The two main 
reasons for doing that is to use the powerful tools developed in the discrete-time framework (see 
for example the references 0, [U IS E3] ) and to avoid working within the continuous-time context 
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and the associated infinitesimal generator, which in many situations has its domain difficult to be 
characterized. For a discounted cost case, the approach adopted by M.H.A. Davis is very natural 
since the key idea is to re-write the integral cost as a sum of integrals between two consecutive 
jump times of the PDMP and, by doing this, naturally obtaining the one step cost function for 
the discrete-time Markov decision model. However, this decomposition for the long run average 
cost is not possible to be done. Our first main result is to propose another approach for obtaining 
an optimality equation for the long run average cost. It is shown that (see Theorem 14. ip if there 
exist a measurable function h, a parameter p and a measurable selector satisfying a discrete-time 
optimality equation related to the embedded Markov chain given by the post-jump location of the 
PDMP, and also that an extra condition involving the function h is verified then an optimal control 
can be obtained from the measurable selector and p is the optimal cost. 

Our second main result (see Theorem 16 . 2[) is to remove the hypothesis of the existence of a 
measurable selector mentioned in the previous theorem and in fact, to guarantee the existence of 
a feedback measurable selector (that is, a selector that depends on the present value of the state 
variable, see Remark 13. IIP provided that the function h and parameter p satisfy the optimality 
equation. This is done by establishing a link (see the proof Theorem I6.4p between the discrete- 
time optimality equation and an integro-differential equation (using the weaker concept of absolute 
continuity along the flow of the value function). The common approach for the existence of a 
measurable selector is to impose semicontinuity properties of the cost function and to introduce 
the class of relaxed controls to get a compactness property for the action space. By doing this one 
obtains an existence result but within the class of relaxed controls. However, what is desired is to 
show the existence of an optimal control in the class of ordinary controls. Combining the existence 
result within the class of relaxed controls with the connection between the integro-differential 
equation and the discrete-time equation we can show that the optimal control is non-relaxed and 
in fact it is an ordinary feedback control. 

In general it is a hard task to get the equality in the solution of the discrete-time optimality 
equation and verify the extra condition. A common approach to avoid this is to consider an 
inequality instead of equality for the optimality equation, and to use an Abelian result to get the 
reverse inequality (see for instance [13]). Our last main result is to obtain some sufficient conditions, 
based on the value function of the discounted control problems, that guarantee the existence of a 
solution for the discrete-time optimality inequality using the so-called vanishing approach (see [IB] , 
page 83). Combining this result with the link between the integro-differential equation and the 
discrete-time equation we obtain the existence of an ordinary optimal feedback control for the long 
run average cost (see Theorem 18. 5p . In order to do that we need first to establish an optimality 
equation for the discounted control problem. 

A closely related paper to ours, but considering the discounted control case, is the paper by 
Forwick, Schal, and Schmitz [13j, which also considers unbounded costs and relaxed controls^ and 
obtain sufficient conditions for the existence of ordinary feedback controls. However, in [13|] the 
authors do not consider the long run average cost case neither the related limit problem associated 
to the vanishing approach. Besides, unlike in [13], we consider here boundary jumps and the 
control action space depending on the state variable. Note however that control on the flow is not 
considered here, while it was studied in [13] • Finally it is worth mentioning that the authors are 
studying in a companion paper the important question of deriving sufficient stability conditions 



(like those presented in [a], ll|]) under which the conditions on the discounted value function 



used in the vanishing approach are satisfied, tracing a parallel with the discrete-time case (see, for 
instance, [15l . 

0). 

The paper is organized in the following way. In section [2] we introduce some notation, basic 
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assumptions, and the control problems to be considered. The definition of the ordinary and relaxed 
control spaces as well as some operators required for characterizing the optimality equation are 
presented in section O This section presents several technical measurability results that will be 
required throughout the paper. The first main result in presented in section 01 Theorem 14.14 which 
obtains an optimality equation for the long run average cost in terms of a discrete-time optimality 
equation related to the embedded Markov chain given by the post-jump location of the PDMP, and 
an additional condition. In section [5] we introduce some continuity assumptions on the parameters 
in order to get some convergence and lower semicontinuity results. In section [6] we derive sufficient 
conditions for the existence of an ordinary feedback optimal control by establishing a link between 
the discrete-time optimality equation and an integro-differential equation using the concept of 
absolute continuity of the value function along the flow (see Theorems 16.21 and I6.4p . Section [7] 
considers the discounted optimal control problem and derive an optimality equation. Our final 
main result is presented in section [8] with some sufficient conditions for the existence of a solution 
for the optimality inequality and an ordinary optimal feedback control for the long run average 
cost using the so-called vanishing discount approach (see Theorem 18. 5p . In order to facilitate the 
reading of the paper several proofs of some technical results are presented in the appendix. 

2 Notation and assumptions 

In this section we present some standard notation and some basic definitions related to the motion 
of a PDMP {X(t)}, and the control problems we will consider throughout the paper. For further 
details and properties the reader is referred to [g]. The following notation will be used in this paper: 

• R denotes the set of real numbers, R+ the set of positive real numbers and R rf the d- 
dimensional euclidian space. 

• rj denotes the Lebesgue measure on R. 

• For X a metric spaces, we denote B(X) as the cr-algebra generated by the open sets of X. 
M.{X) (respectively, V{X)) denotes the set of all finite (respectively probability) measures 
on (X,B{X)). 

• Let X and Y be metric spaces. The set of all Borel measurable (respectively bounded) 
functions from X into Y is denoted by M.(X;Y) (respectively M(X;Y)). Moreover, for nota- 
tional simplicity M(X) (respectively M(X), M(X)+, JB(X)+) denotes M(X;R) (respectively 
B(X;R), M(X;R+), 1(X;R+)). C(X) denotes the set of continuous functions from X to R. 
For h £ M(-E), h + (respectively h~) denotes the positive (respectively, negtive) part of h. 

To get a better picture of the motion of a PDMP we first present its definition without any control 
variable. Let E be an open subset of R n , dE its boundary, and E its closure. A PDMP is determined 
by its local characteristics (<p,X,Q). The main assumptions and related definitions on these three 
parameters are presented below: 

• the flow <j)(x, t) is a function <f> : R n x R + — ► R n continuous in (x, t) and such that 

<f>(x,t + s) = (f>(<t>(x,t),s). (1) 
For each x G E the time the flow takes to reach the boundary starting from x is defined as 

U(x) = inf{i > : <j>(x,t) G dE). 
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For x G E such that t*(x) = oo (that is, the flow starting from x never touches the boundary), we 
set <p(x, £*(#)) = A, where A is a fixed point in dE. 



Some results that will be derived along the paper will be written in terms of the properties along 
the flow 4>(x, t). In particular we define the following space of functions absolutely continuous along 
the flow with limit towards the boundary: 

M ac (E) = {g G M(E);g((j)(x,t)) : [0 i-> R is absolutely continuous for each x G E 

and whenever t*(x) < oo the limit lim g(6(x,t)) exists}. 

t-*U(x) 

For g G Wl ac (E) and z G dE for which there exists xeE such that z = 4>{x, t*(x)) where t*{x) < oo 

we define g(z) = lim g(<J)(x,t)) (note that the limit exists by assumption). As shown in Lemma 

t— >t*(x) 

IPl for g G M ac (E) there exists a function Xg G M(£) such that for all x G E and f G [0,t*(x)) 

g(4>(x,t)) - g{x) = / Affli(^(ar,s))da. 

JO 

• the jump rate A : E — > R+ which is assumed to be a measurable function satisfying: (Vx G £7) 

(3e > 0) such that / X(c/)(x, s))ds < oo. 
Jo 

• the post-jump location kernel Q : E x B{E) — > [0, 1] which is a transition measure satisfying the 
following property: (Vx G E) Q(x,E — {x}) = 1. 

From these characteristics, it can be shown [8j, p. 62-66] that there exists a filtered probability space 
(p,, J 7 , {Ft}, {Px}xge) such that the motion of the process {X(t)} starting from a point x G E may 
be constructed as follows. Take a random variable T\ such that 



Px(Ti > t) 

and t G [0, t*(x)[ 



e ~A(x,t) for t < t ^ x j 
fort>t*(x) 



A(x,t) = I \(<f>(x,s))ds. 
Jo 

If T\ generated according to the above probability is equal to infinity, then for t G R+, X(t) = 
4>(x,t). Otherwise select independently an E- valued random variable (labelled X\) having distri- 
bution Q(4>(x,Ti), .). The trajectory of {X(t)} starting at x, for t < T\ , is given by 



X(t) 



4>(x, t) for t < T\, 
Xi fort = Ti. 



Starting from X(T\) = X\, we now select the next inter-jump time T<i — T\ and post-jump location 
X(T2) = X2 in a similar way. This gives a strong Markov process {X(t)} with jump times {£fc} fcgN 
(where To = 0). 

We present next the control problems and some basic assumptions we will consider throughout 
the paper. We suppose from now on that the local characteristics A and Q depend on a control 
action u G U where U is a Borel space, in the following way: 
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• AG M(E x U)+. 

• Q is a stochastic kernel on E given £xD. 

For each x £ Ewe define the subsets U(x) of U as the set of feasible control actions that can be 
taken when the state process is in x G E, that is, the control action that will be applied to A and Q 
must belong to U(x). The following assumptions, based on the standard theory of Markov decision 
processes (see [3]), will be made throughout the paper: 

Assumption 2.1 For all x £ E, U(x) is a compact subspace o/U. 

Assumption 2.2 The set K = {(a, a) : x G E,a G U(x)} is a Borel subset of E x U. 

The following assumption will also be required along the paper: 

Assumption 2.3 For all x G E, and t G [0, t*(x)), I sup \{4>(x , s) , a) ds < oo. 

JO o6U(0(a!,«)) 

/•I* (a) 

If t*{x) < oo i/ien / sup A(<^>(a;, s), a) ds < oo. 

•/ o6U(<4(a;,s)) 



We present next the definition of an admissible control strategy and the associated motion of the 
controlled process. A control policy U is a pair of functions (u,u d ) G M(Nx£xM+;U)xM(Nx£;U) 
satisfying u(n,x,t) 6 V (cj)(x , t)) , and uq{h,x) G U(<^>(x, for all (n,x,t) fNx J?x R + . 

The class of admissible control strategy will be denoted by U. 

Given a control strategy U = (u, ug), one describe the motion of the piecewise deterministic process 
X(t) in the following manner. Define Tq = and X(0) = x. Assume that the process {X(t)} is 
located at Z n at the n th jump time T n then select a random variable S n having distribution 

F(t) = 1 - I{ t<U (z n )}£~ ■^ XWZn ' t) ' u{n,ZnyS))ds ■ 
Define T n+1 =T n + S n and for t € [T n ,T n+1 ), X(t) = 4>(Z n ,t - T n ). 

Let Z n+ \ a random variable having distribution Q((j)(Z n , T n+ \), u(n, Z n , S n )); .) if (f)(Z n ,T n +i) € 

or Q(cj)(Z n ,T n+1 ),u d (n, Z n ); .) if 0(Z n ,T n+ i) G 

At time T n+ i, the process {X(t}} is defined by X(T n+ i) = Z n+ \. 

Now we give a more precise definition of the controlled piecewise deterministic Markov process 
described above. Consider the state space E = E x E x M + x N. For a control policy U = (u, uq) 
let us introduce the following parameters for x = (x, z, s, n) E E: 

• the flow <f>(x, t) = (<f>(x, t), z,s + t,n). 

• the jump rate X u (x) = X(x, u(n, z, s)). 

• the transition measure 



Q u (x,A x B x {0} x {n + 1}) 



Q(i,w(n,2,s));inB) if x G E, 
Q(x,u d (n,z);AnB) if x G dE, 



From [8j, section 25], it can be shown that for any control strategy U = (u,Ug) G U there exists 
a filtered probability space (Q, J 7 , {Tt}, {P^} ie ^) such that the piecewise deterministic Markov 

process {X u (t)} with local characteristics ((f), \ u , Q u ) may be constructed as follows. For notational 



for A and B in B(E). 
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simplicity the probability will be denoted by PV fc s for xq = (x, x, 0, k) G Take a random 



variable Ti such that 



P^iTi > t) = 



■ e -A u (x,k,t) for t < t ^ 

for t > U(x) 



where for x £ E and t G [0, 



A (x,k,t) = / \((f>(x, s), u(k, x, s))ds. 
Jo 

If Xi is equal to infinity, then for t G E + , X^(t) = (<^>(x, i), cc, i, 0) . Otherwise select independently 
an .E-valued random variable (labelled X^) having distribution 



Fid*? eAxBx{0}x{k + ljKli}) 



x,T 1 ),u{k,x,T 1 ));Af\B) if <f>[x,T{) G E, 
x,T 1 ),u d {k,x);A(lB) if 0(x,Ti) G &E. 



The trajectory of {X u (t)} starting from (x,x,0,k), for t<T\ , is given by 

X u (t) 



(4>(x,t),x,t,0) for i < Ti, 
Xf fort = Ti. 



Starting from X^(Ti) = X^ 7 , we now select the next inter-jump time T2—T1 and post-jump location 
X u {T2) = Xrf in a similar way. 

Let us define the components of the PDMP {X u (t)} by X u (t) = (X(t), Z(t),r(t), N(t)) . From 
the previous construction of the PDMP {X u (t)}, it is easy to see that X(t) corresponds to the 
trajectory of the system, Z(t) is the value of X(t) at the last jump time before t, r(i) is time 
elapsed between the last jump and time t, and N(t) is the number of jumps of the process {X(t)} 
at time t. 

As in Davis [a], we consider the following assumption to avoid any accumulation point of the jump 
times: 



Assumption 2.4 For any x G E, U = (u, uq) G U, and t>0, e¥ 



(*,o) 



.i=l 



< OO 



The costs of our control problem will contain two terms, a running cost / and a boundary cost r, 
satisfying the following properties: 

Assumption 2.5 / G M(E x U)+. 
Assumption 2.6 r G M(dE x U)+. 

The long-run average cost we want to minimize over U is given by: 

1 



A(U,x) 



lim =-EV xQ - ) 



f(X(s),u(N(s),Z(s),r(s)))ds 



+ / r(X(s-),u d (N(s-),Z(s-)))dp*( S ) 
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where p*(t) = I{Ti<t}I{X(Ti-)edE} counts the number of times the process hits the boundary 



i=i 



up to time t, and we set 



Ja{x) 



inf A(U, x) 



(3) 



For the a discounted case, with a > 0, the cost we want to minimize is given by: 
V a (U,x)=E l 



f(X(s),u(N(s),Z( S ),r( S )))ds 

/•oo 

+ / e - as r{X(s-), Ud (N(s-),Z(s-)))dp*(t 
Jo 



and we set 



JS(x) = inf V a (U,x). 
U&A 



(4) 
(5) 



We also consider a truncated version of problem 



V«(U,x)=E 



u 

(x,0) 







defined, for each m = 0,1, 
f{X(s),u(N( S ),Z( S ),r( S )))ds 



as 



+ 



Sr{X(s-),u d (N(s-),Z(s-)))dp*(i 



(6) 



We need the following assumption, to avoid infinite costs for the discounted case. 
Assumption 2.7 For all a > and all x G E, J%{x) < oo. 

It is clear that for all x £ E, < inf V°L(U,x) < J%ix) < oo. 

U&A 

Remark 2.8 There is no loss of generality in assuming that U is compact. Indeed, if V is a Borel 
space, it follows from Proposition 7 in ]2W that U can be considered as a measurable subset of 
a compact space U' where V(x) is compact in V for x € E. Moreover, by recalling that U(x) 
represents the set of feasible controls in the state x € E, the definition of the control problem will 
not be affected if the functions A, Q, f , and r are extended to V . Therefore, from now on one will 
assume that U is compact. This result will be needed in sub-section \3.1\ 



3 Discrete-time ordinary and relaxed controls 

The class of open loop policies denoted by U has been introduced in the previous section as the 
set of admissible control strategies for a PDMP. As mentioned in the introduction we will study 
in section H] how the original continuous-time control problem can be associated to an optimality 
equation of a discrete-time problem related to the embedded Markov chain given by the post-jump 
location of the PDMP. In this section we first present the definitions of the discrete-time ordinary 
and relaxed control sets used in the formulation of the optimality equation of the discrete-time 
Markov control problem as well as the characterization of some topological properties of these sets. 
In particular, by using a result of the theory of multifunctions (see the book by Castaing and 
Valadier [3]), it is shown that the set of relaxed controls is compact. In the sequel we present some 
important operators associated to the optimality equation of the discrete-time problem as well as 
some measur ability properties. 
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3.1 Relaxed and ordinary control 

We present in this sub-section the set of discrete-time relaxed controls and the subset of or- 
dinary controls. Consider the Banach spaces L 1 (R + ; C(U)) and L°° (R + ; M(U)) where C(U) is 
equipped with the topology of uniform convergence and A4(V) is equipped with the weak* topol- 
ogy a ( M. (U), C(U)). Let V r (respectively V r (x) for x G E) be the set of all 77-measurable functions 
/i defined on R + with value in 7HU) such that /J,(t,U) = 1 rj-a.e. (respectively /j,(t, U(4>(x, t))) = 1 
r/-a.e.). From Theorem V-2 in |3j, it follows that V r (respectively V r (x) for x G E) are compact 
sets with respect to the weak* topology a(L°°(R + ; M(V)),L 1 (R + ; C(U))). Moreover, from Bishop's 
Theorem (see Theorem 1.3.11 in [23]), there is a metric such for all x G E, V r (x) is a compact set 
of the Borel set V r . Note that a sequence (/%)„ €N in V r {x) converges to \i if and only if 



lim / / g(t,u)/j, n (t,du)dt = / / g(t,u)/j,(t,du)dt, (7) 

(x,t)) Jr + Jv((f>(x,t)) 



for all g G L 1 (R + ; C(U)). 

Therefore, the set of relaxed controls are defined as follows. For x G E, 

Y r (x) = V r {x) x V(U(<f)(x,t*(x)))), 
Y r = V xP(U). 

The set of ordinary controls, denoted by V (respectively Y(x) for x G E), is defined as above except 
that it is composed of deterministic functions instead of probability measures. More specifically we 
have 

V(x) = jV G M(R_|_, U) : (Vt G R+), v{t) G U(<fi(x, , 

Y(x) = V(x) xU(^(x,t*(x))), 
V = M(M+,U)xU. 

Consequently, the set of ordinary controls is a subset of the set of relaxed controls V r (respectively 
Y r (x) for x G E) by identifying any control action n G U with the Dirac measure concentrated on 
u. Thus we can write that V C V r (respectively Y(x) C Y r (x) for x G E) and from now on we will 
consider that V (respectively Y(x) for x G E) will be endowed with the topology generated by V r . 

The necessity to introduce the class of relaxed control Y r is justified by the fact that in general 
there does not exist a topology for which V and Y(x) are compact sets. However from the previous 
construction, it follows that V r and Y r (x) are compact sets. 

We present next a definition that will be useful in the next sections. 
Definition 3.1 For any x G E, t G {0,t*(x)) ; and = (/U,/xg) G Y r (x), define 

[Q] t = (fi(. + t),ii d ). (8) 

Clearly, [0] G Y r ((j)(x, t)). 



As in [16|], page 14, we need that the set of feasible state/relaxed-control pairs is a measurable 
subset of B{E) x B(Y r ), that is, we need the following assumption. 
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Assumption 3.2 

K = {(x,@): OeV r (x),x B{E) x B(Y r ) 



We present a sufficient condition, based on the continuity of the sets U(x), to ensure that assumption 
13.21 holds. The proof is presented in the appendix. 

Proposition 3.3 Assumption \3.2\ is satisfied if for all convergent sequence {£ n }neN in E 

u(x) = n u u(x m ) = u n u(x m ), 

where lim x n = x. 

n^oo 

3.2 Discrete-time operators and measurability properties 

In this sub-section we present some important operators associated to the optimality equation of 
the discrete-time problem as well as some measurability properties. 

We consider the following notation for x G E, y G E and ji G V(U), h E M(i?) + , and 
w G M(E x U)+: 

Qh(x,fi)= / / h(z)Q(x,u; dz)fi(du), \Qh(x,^)= / X(x,u) / h(z)Q(x,u; dz)fi(du), 
Jv Je Jv J e 

A M (y, t) = / / \(4>(y, s),u)/j,(du)ds, w(x,fi)= / w(x,u)[i(du). 

Jo Jv Jv 

The following operators will be associated to the optimality equations of the discrete-time problems 
that will be presented in the next sections. For = (/U, fjLg) G V r , h G M.(E), a G R, let us introduce 
the following stochastic kernel defined onEx B(E) by 

ft* (x) 

G a {x,0;A) = / e - QS - AM ^ s )AQI A (0(x,s),^( S ))d S 



+e -at,(x)-A^M-)) Q ^ t*(x)),» d ; A), (9) 

for all (x, A) G E x and so for h G M(E)+, we define G a h{x, 0) = / h(y)G a (x, 0; dy). For 

x G £7, 6 = G V r , v G M(£7 x U)+ w G M(dE x U), a G R, introduce 

L a t>(x,0) = / e- as - A " {x > s) v(<j){x,s),ti{s))ds, (10) 



F aW (x,9) = e-^-^^wi^xM^^d)- (11) 

For ft G M(E) (respectively, v G M(E x U)), G a h(x,Q) = G a h + (x,&) - G a h~(x,Q) (respectively 
L a v(x, 0) = L a v + (x, 0) — L a v~(x, 0)) provided the difference has a meaning. It will be useful in 
the sequel to define the function C a (x, 0) as follows: C a (x,@) = L a lExv(x,&)- In particular for 
a = we write for simplicity Go = G, Lq = L, Hq = H, Co = C. 

The next proposition presents some important measurability properties of the operators G a , L a , 
and H a (defined in equations ([9]), (fTOj) . and (fTTj) ) and its proof can be found in the appendix. 

Proposition 3.4 Let a G R+, g G M(E') be bounded from below, w\ G M.(E x U) 6e bounded from 
below, and u>2 G M.(dE x U). T/ien the mappings G a g(x,Q), L a w\{x,Q), and H a W2(x,Q) defined 
on E x V r with values in R are B(E x V r )-measurable. 
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We present now the definitions of the one-stage optimization operators. 
Definition 3.5 For a G R+, p G R+, and g G M.(E) bounded from below, define 

i) the (ordinary) one-stage optimization operator by 

T a {p,g)(x) = inf \-pC a {x,T) + L a f(x,T) + H a r(x,T) + G a g(x,T)\. (12) 

ii) the relaxed one-stage optimization operator by 

K a (p,g)(x) = inf \-pC a (x,@)+L a f(x,e) + H a r{x,@)+G a g(x,e)\. (13) 

In particular for a = we write for simplicity % = T, and TZo = TZ. 

Let us introduce the following sets of measurable selectors associated to (U(x)) eE (respectively 
(Y(x)) xeE , (W(x)) xeE ): 

Su = {ueM(E,U):(VxG£),u(i)GU(x)}, 
S Y = {{v,v d )€M{E,V):(yxeE),(v(x),v a {x))€V(x)}, 
S Y r = {(p,p 9 ) G M(E,V r ) : (Vx G E), (p(x), p 9 (x)) G V r (x)}. 

Remark 3.6 The set Su characterizes the control law u(x) that only depends on the value of the 
state variable x. On the other hand (v, vq) G Sy characterizes an ordinary control law for the control 
problem associated to the one-stage optimization operator. Indeed, starting from x, it defines the 
control law for all t G [0,t*(a;)) through the function v(x,t) and at t = t^(x) (ift*(x) < oo) through 
uq. Finally (p, pg) G Syr characterizes a relaxed control law for the control problem associated to 
the relaxed one-stage optimization operator. Since starting from x, it defines a probability over the 
feasible control actions for all t G [0,t*(x)) through the probability measure p(x,t) and at t = t*(x) 
(if t*(x) < oo) through the probability measure pg. 

For a G R+, p G R+, and g G M(E) bounded from below, the one-stage optimization problem 
associated to the operator T a (p,g), respectively TZ a (p,g), consists of finding a measurable selector 
T G Sy, respectively G Syr such that for all x G E 

T a {p,g)(x) = -pC a (x,T(x)) + L a f(x,T(x)) + H a r(x,T(x)) + G a g(x,T(x)), 

respectively 

K a {p,g)(x) = -pC a (x,e(x)) + L a f(x,e(x)) + H a r(x,e(x)) + G a g(x,e(x)). 

Finally we conclude this section by showing that there exist two natural mappings from Su to Sy 
and from Su to U 

Definition 3.7 For u G Su, define the mapping : x — > (u(c/)(x, .)),u((f>(x,t*(x)))) of the space 
E into V 
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Proposition 3.8 If u E Sjj then G Sy. 

Proof: From Lemma A. 3 in [l3| and item (i) of Lemma 3 in [25] it follows that the mapping 
x — > u(<p(x,.)) of the space E into M(R+,U) is measurable. Moreover, for all (x,t) £ Ex M + , 
u(4>(x,t)) G U(</>(x, i)). Therefore, belongs to Sy. □ 

Definition 3.9 For u G 5u, define the mapping : (n,x,t) — > m(^(x, t)), u(<p(x, i*(x)))) of the 
space Nxi?xl + into U x U. 

Proposition 3.10 If u £ Su then G W. 

Proof: This is a straightforward consequence of the measurability properties of it and </> and the 
fact that u(x) G U(x). □ 

Remark 3.11 The measurable selectors of the kind as in Definition \3. 7| are called feedback 
measurable selectors in the class Sy C Syr and the control strategies of the kind XJ$ as in definition 
\3.iA are called feedback control strategies in the class U. 

4 Discrete-time optimality equation for the average control 

In this section we obtain an optimality equation for the long run average cost problem defined in 
equation (|3|) in terms of a discrete-time optimality equation related to the embedded Markov chain 
given by the post-jump location of the PDMP, and an additional condition on a limit over the 
solution of the optimality equation divided by the time t. Notice that in this section we will be 
assuming that there is an optimal solution for the one-stage optimization problem, and therefore 
we will be dealing with the ordinary action sets (that is, V(x)). 



Theorem 4.1 Suppose that there exists a pair (p,h) G 
satisfying the following discrete-time optimality equation 



T(p,h)(x) 



- x M(.E') with h bounded from below 
h(x), (14) 



and for all U 



lim - lim e¥ t(] \ 



h(X(tAT m )) 



0. 



(15) 



Moreover, assume that there exists a solution to the one-stage optimization problem associated to 
T(p, h), that is, the existence of an optimal measurable selector T = (7,7,9) in Sy such that for all 
x G E 



T(p,h)(x) 



-p£(x, f (x)) + Lf(x, f (x)) + Hr(x, f (x)) + Gh(x,f(x)), 



(16) 



Define the control strategy U by (u,uq) with u(n,x,t) = j(x,t), ug(n,x) = 7a(x) for (n,x,t) G 
N x E x M + . Then U belongs to U and it is optimal. Moreover, 



P = J A (x)=A(U,x). 
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The proof of this theorem is presented at the end of this section. 



In Theorem 14. II notice that equation (|14h can be seen as the optimality equation of a discrete-time 
problem related to the embedded Markov chain given by the post-jump location of the PDMP with 
transition kernel G, and equation (|15p as an additional technical condition. 



Notice that in general, one cannot guarantee the existence of an optimal measurable selector for 
the optimality equation (|14|) without compactness conditions. This problem of existence will be 
considered in the next section. 

In order to prove the previous theorem we first need to present several intermediate results that 
will also be used in the remainder of the paper. Notice that these results will be written in terms 
of an extra parameter a > that will be useful for the discounted control problem analyzed in 
sections [7] and El The proofs of these intermediate results can be found in the appendix. 

The next proposition presents some important properties of the one-stage optimality equation (see 
equations (fT7]) . ([18])). It is shown that the solution of the one-stage optimality equation has a 
special time representation (see equation (I19p ). As a consequence it follows that it is absolutely 
continuous along trajectories with limit on the boundary (that is, it belongs to M ac (£")). 

Proposition 4.2 Let p G R+ and h G ~M(E) be bounded from below. For a > and x G E define 



w(x) = T a (p,h)(x). 

Assume that w G M(i£) and there exists T G <Sy such that 

w(x) = -p£ a (x, T(x)) + L a f(x, T(x)) + H a r(x, T(x)) + G a h(x, T(x)). 
Then w G M ac (E) and for all x G E, and t G [0, t*{x)), 

j-OS-AT'W (x,s) 



(17) 



(18) 



w\x) 



~P + f(4>{x, s),j(x, s)) + XQh(<p(x, s),j(x, s)) 



ds 



-at-K"i( x ~>(x,t)^ 



inf 



-as—A"(x,s) 







-P + f{<t>{x, s), u(s)) + \Qh(<j)(x, s),v(s)) 



ds 



+ e 



-at-A"(x,t) 



w((f>(x,t)) 



(19) 



(20) 



where f(x) = (^{x), 79 (x)) . 



The next two propositions deal with two inequalities of opposite directions for the one-stage opti- 
mality equation. Roughly speaking these two results show that if h is a solution for a one-stage 
optimality inequality (see pip or (I23p ) then this inequality is preserved, in one case for any control 
strategy and in the other case for a specific control strategy, along the jump time iterations for a 
cost conveniently defined, see equations (|22|) or (f24l) . 
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Proposition 4.3 Let h G M.(E) be bounded from below. For p G M+ and a G M + , assume that 
T a (p, h) G M.(E) and there exists T = (7, jg) G Sy such that for all x G E 

h{x) < T a (p, h)(x) = -p£ a (x, f(x)) + L a f(x, f (x)) + H a r(x, f (x)) + G a /i(x, f (x)). (21) 

For {7 £ W, ra £ N, and (t, x, fc) G M+ xfixN, de/me 



J^(t,x,k) =E t 



u 

(x,k) 







/(X( S ),u(iV( S ),Z( S ),r( S ))) -p 



ds 



tAT„ 



- I e- as r(X(s-),u d (N(s),X(s-)))dp*(s)+e-^ tAT ^T a (p,h)(X(tAT m )) 
10 



Then for all m G N, and (t, x, k) G M + x E x N, 

jU(t,x,k) > h(x). 



(22) 



The next proposition considers the reverse inequality. 

Proposition 4.4 Let h G M.(E) be bounded from below. For p G M+ and a G K+, assume that 
T a (p, h) G M(F) and there exists T = (7, 79) G <SV snc/i i/iai for all x E E 

h(x) >T a (p,h)(x) =-pC a (x,T(x)) + L a f(x,r(x)) + H a r(x,r(x)) + G a h(x,r(x)). (23) 

Then for U defined by (u, uq) with u(n, x, t) = ^(x, t), and uq{u, x) = 7o(x) for (n,x,t) £NxJ?x 
K+, we have that U belongs to U. Moreover, defining 

rt/\T„. 



J^(t,x,k) =E t 



(x,k) 







f(X(s),u(N(s),Z( S ),r(s))) -p 



ds 



tAT„ 



- I e- as r(X(s-),u d (N(s),X( S -)))dp*(s)+e~^ tAT ^T a (p,h)(X(tAT m )) 
'0 

we have, for all m G N ; (t, x, k) G M+ x£xN, i/iai 

jV(t,x,k)<h(x). (24) 

Combining the previous propositions with a = we get the proof of Theorem 14.11 
Proof of Theorem 14.11 From Proposition 14.31 it follows that 



E: 



(*,0) 



tAT m rtAT m 

f{X(s),u(N( S ),Z(s),r(s)))ds+ / r(X(s-),u d (N(s),X(s-)))dp*(s) 



+ E, 



(*,0) 



h(X(tAT m )) 



> E 



u 

(x,0) 



p[tAT m ] +h(x) 



Consequently we have, from assumption 12.41 (which implies that T m — > 00 P u a.s.), that 



lim -EV x0) 



1 f{X(s), u(N(s), Z(s),t(s))) ds + f r {X(s-), u 9 (N(s), X(s-)))dp*(s) 
Jo 



1 



+ lim - lim EfL^ 



h(X(tAT m )) 
showing that p < Ja( x )-> by using equation (fT5l) . 



> 



From Proposition 14.41 the control strategy U defined by (u,ug) with u(n,x,t) = 7(2;, i), uq{u,x) = 
7a(x) for (n, x, t) G N x E x IR + belongs to IA. Combining equations (|15p and ()24p we obtain that 
p > ^4.(^5 2;) completing the proof. □ 
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5 Convergence and lower semicontinuity properties 



In the previous section we assumed the existence of an ordinary optimal measurable selector for 
the one-stage optimization problem associated to T(p,h) (see equation (fTo| ). for (p,h) satisfying 
the optimality equation T(p,h)(x) = h(x). In the next sections we will suppress this hypothesis. 
In order to do that we need to consider relaxed controls, so that we can take advantage of the 
compactness property of the sets Y r (x) and V r as presented in sectional Note however that we 
also need the cost function to be lower semicontinuous. Thus in this section we present the as- 
sumptions and results that will guarantee some convergence and lower semicontinuity properties 
of the operators G a , L a , and H a that appear in the one-stage optimization operators with respect 
to the topology defined in equation (J7|). Combining the compactness of the sets V(x) with the 
lower semicontinuity of the operators Gg_^L a , and H a we can use the measurable selector theo- 



rem as presented in Proposition D.5 of [16f] to get the existence of a relaxed optimal control and 
measurability of the one-stage optimization equation. Moreover in parallel we get some important 
convergence properties that will be applied with the vanishing approach in section [HJ 

From now on we will consider the following assumptions. 

Assumption 5.1 For each x G E, X(x, .) : U(x) i— ► R+ is continuous. 

Assumption 5.2 There exists a sequence of measurable functions (/j)ieN inWl(E xU) + such that 
for all y G E, fj(y, .) | f(y, .) as j -> oo and fj(y, .) G C(U(y)). 

Assumption 5.3 There exists a sequence of measurable functions (fj)j^ in M.(dE x U) + such 
that for all z G dE, rj(z, .) j r(z, .) as j — > co and rj(z, .) G C(U(z)). 

Assumption 5.4 For all x G E and g G M(E), Qg(x, .) : U(x) i— > R is continuous. 

Assumption 5.5 There exists £ G M.(E) + , such that 
a) X(y, a) > £(y) for all y G E and a G U(y), 



b) 



rU(x) 

/ e ~ fo m*,s))ds dt < Ki<00 f or a u xeE , 
Jo 



ft, (x) 

c) / e -fomx,s))ds gup f^ Xit ),a)dt < oo. 

JO a&]{4>(x,t)) 

Remark 5.6 A consequence of Assumption [5751 b) is that lim e~ at ~fo €(<t>( x > s )) ds = o ? for any a G 
R + and i££ with = +oo. Therefore, for any x G E with = +oo, A G B(E), a G R+, 

rU (x) 

6 = (p, p d ) g Y r {x), w G M(dE x U), G a {x, Q;A)= / e -as-A^(x,s) X Qi A ^( Xj s ),p{s))ds, and 
H a w(x,Q) = 0. 



o 



The next proposition presents convergence results of the operators G a , L a , and H a with respect to 
the topology defined in equation ([7]). Note that the convergence is taken not only with respect to a 
sequence of controls but also with respect to some functions and the parameter a. This is justified 
by the fact we will need this convergence for the vanishing approach in section [8j The proof of the 
proposition is in the appendix. 
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Proposition 5.7 Consider a € M+ and a non increasing sequence of positive numbers {a^}, j. 
a, a sequence of functions h ak € M.(E) uniformly bounded from below by a positive constant (that 
is, h ak {y) > —Kh for all y € E). Set h = lim h ak . For x £ E, consider Q n = (p n , Hd,n) 6 V r (x) 

fc— >oo 

and = (//, /j,q) € Y r (x) such that O n — ► 0. We have the following results: 

a) lim C an (x,@ n ) = £(x,Q). 

n— >oo 

b) lim L Q J(x,e n )>Lf(x,Q). 

n^oo 

c) lim H an r(x,Q n ) > Hr(x,Q). 

n— >oo 

d) lim G Qn /i an (x,9 n ) > Gh(x,@). 

n— >oo 

The lower semicontinuity properties mentioned at the beginning of this section follow easily from 
this proposition as stated in the next corollary. 

Corollary 5.8 Consider h € M(E) bounded from below. We have the following results: 

a) C a (x,Q) is continuous on V r (x). 

b) L a f(x,Q) is lower semicontinuous onV r (x). 

c) H a r(x,Q) is lower semicontinuous onY r (x). 

d) G a h(x,Q) is lower semicontinuous onW r (x). 

Proof: By taking = a > 0, h ak = h in Proposition 15,71 the results follow. □ 

6 Existence of an ordinary optimal feedback control 

The main result of this section is the Theorem 16.21 that strengthens Theorem 14.11 of the previous 
section by only assuming that the discrete-time optimality equation T(p, h) = h has a solution 
in order to ensure the existence of an optimal control strategy for the long run average control 
problem. Moreover, it is shown that this optimal control strategy is in the feedback class and can 
be characterized as in item D3) of the Definition 16.11 These results are obtained by establishing 
a connection (see the proof Theorem I6.4p between the discrete-time optimality equation and an 
integro-differential equation (using the weaker concept of absolute continuity along the flow of the 
value function). The basic idea is to use the set of relaxed controls Y r (x). The advantage of 
considering V r (x) is that it is compact so that, together with the assumptions we have made in 
section [5l we can apply a measurable selector theorem to guarantee the existence of an optimal 
measurable selector (see Proposition 16.31) . The price to pay is that this measurable selector belongs 
to the space of relaxed controls. However we can show that in fact there exists a non-relaxed 
feedback selector for the discrete-time optimality equation T(p, h) = h by establishing a connection 
between the discrete-time optimality equation and the integro-differential equation (see the proof 
of Theorem 16. 4|) . 

Definition 6.1 Consider w £ M.(E) and h 6 M.(E) bounded from below. 
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Dl) Denote by u(w,h) G Sjj the measurable selector satisfying 



inf {f(x, a) — \(x,a) w(x) — Qh(x,a) } 

a<EV(x) L J 

= f(x, u(w, h)(x)) — X(x, u(w, h)(x)) w(x) — Qh(x, u(w, h)(x)) 

inf {r(z, a) + Qh(z, a)} = r(z,u(w, h){z)) + Qh(z,u(w, h)(z)). 

D2) u^(w,h) G Sy is the measurable selector derived from u(w,h) through the definition \3. 
D3) U^w^) 6W is the control strategy derived from u(w,h) through the definition \3.9\ 



The existence of u(w, h) follows from assumptions 15.1115,41 and Theorem 3.3.5 in [16] and the fact 
that U(j,(w, h) G <Sv, and U^(w,h) G U comes from Propositions 13.81 and 13.101 



Theorem 6.2 Suppose that there exists a pair (p, h) G R x M.(E) with h bounded from below 
satisfying the following discrete-time optimality equation 



T(p,h)(x) = h(x), 



and for all U ElA, 



lim - lim EV n) 



h(X(tAT m )) 



0. 



Then h G M ac (.E), the feedback optimal control strategy U^,(h,h) (see item D3) of Definition \6.1\) 
is optimal, and 

p = JA(x)=A{U(h,h),x). 

Proof: The proof of this result is straightforward by combining Theorem l4.1l of the previous section 
and Theorem 16.41 presented below. □ 

The proof of the next proposition is presented in the appendix. It shows the existence of an optimal 
relaxed measurable selector for the relaxed one-stage optimization operator TZ a (p, h)(x) and that 
K a (p,h) G M ac (E). 

Proposition 6.3 Let a > 0, p G M+ and h G M.(E) be bounded from below. For x G E define 
w(x) = TZ a (p, h)(x). Assume that for all x G E, w(x) G M. Then there exists G Syr such that 



w (x) = -pC a (x, 0(x)) + L a f(x, G(x)) + H a r(x, 9(x)) + G a h(x, 6(x)). 
Moreover, w G M. ac (E), and satisfies for all x G E and t G [0, t*(x)) 



(25) 



w{x) 



inf 



-as~K> i {x,s) 



+ e- at - K ^ x ^w{<P{x,t)) 



~P + f{<t>{x, s),n{s)) + \Qh{(j)(x, s),p(s)) 



ds 



~P + f{4>{x, s),p(x, s)) + \Qh(<j)(x, s), p,(x, s)) 



ds 



+ e- at - AKx) ^w(cf>(x,t)), 
where 0(x) = (£t(x), £ig(x)). 



(26) 



(27) 
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The following theorem shows the existence of a feedback measurable selector for the one-stage 
optimization problems associated to T a (p, h) and 1Z a (p, h). Its proof is presented in the appendix. 

Theorem 6.4 Let a > 0, p G M+ and h G M.(E) be bounded from below. For x G E define 

w(x) =n a (p,h)(x), (28) 

and suppose that w(x) G R for all x G E. Then w G M ac (i?) and i/ie feedback measurable selec- 
tor u${w,h) G 5y (see item D2) of Definition flO) satisfies the following one-stage optimization 
problems: 

K a (p, h) (x) = T a (p, h) (x) 

= -pC a (x,u (j) (w,h)(x)) + L a f(x,u ( f,(w,h)(x)) + H a r(x,u (f) (w,h)(x)) 

+G a h(x,u (P (w,h)(x)). (29) 

7 Optimality equation for the discounted case 

In this section we consider the discounted optimal control problem ([5]) and, under the assumptions 
made in the previous sections, we derive an optimality equation for this problem. As usual in this 
kind of problem we characterize first the optimality equation for the truncated on the jump times 
T m problems (|6|) and then take the limit as m — > oo. 

Throughout this section we consider a > fixed. For any g £ M.(E) + , we set Wg as the function 
on E defined as 

Wg(x) = TZ a (0,g)(x), (30) 

for x G E. The following proposition is an immediate consequence of the results derived in the 
previous section. 

Proposition 7.1 For g E M.(E) + consider w = Wg and suppose that for all x E E, w(x) £ R. 
Then w € M(£') + and u^(w,g) G 5y (see item D2) of Definition \6.1\) satisfies 

w(x) = L a f(x,u ( f > {w,g)(x)) + H a r{x,u 4 ,{w,g)(x)) + G a g(x,u <t) (w,g){x)). (31) 

Proof: From Theorem 16.41 we obtain the first equality and that u^(w,g) satisfies equation pip . 
Now applying Proposition 13.41 and by using the fact that / £ M(E x U) + and r G M(dE x U) + , 
we obtain that Wg G M(£) + . □ 

Define the sequence of functions (f m ) m eN as 

v m+1 = Wv m , v Q = 0, (32) 

We have the following proposition. 

Proposition 7.2 For all x G E and m G N we have that v m (x) = inf T>^ n {U,x). 

Proof : It follows from the same lines as the proof of the Propositions 14.31 and 14.41 □ 

Since v m (x) = inf V^U^x) < J%{x\ the functions v m G M(i?) + and are non-decreasing. Con- 

sequently, there exists v G M(£') + such that v m f v, and it follows that v < J£. We need the 
following propositions: 



18 



Proposition 7.3 If h € M(E) + is such that h(x) > Wh{x) then h(x) > 3%(x). 

Proof: By using Theorem 16.41 with p = 0, we obtain that there exists u$ G S\ such that 

h(x) > T a (0, g)(x) = L a f(x, u^x)) + H a r(x, u<j>{x)) + G a h(x, u^(x)). 

Define w(x) = L a f(x,u ( f > (x)) + H a r(x,u < f ) (x)) + G a h(x ,u^{x)) . Clearly w(x) > 0. Moreover, the 
hypotheses of Proposition 14.41 are satisfied with p = 0. Consequently, it follows that there exists 
U € U such that for all m G N, (f , x, k) G M + x E x N, 

"tAT m 



E 



u 

(x,k) 



f{X(s),u(N( S ),Z(s),r( S ))) 



ds 



From assumption 



rtAT m 

+ / e 
J o 

< h(x). 
(which implies that T n 



*r(X(s-),u d (N(s),X(s-)))dp*(s) 



co P u a.s.), we have that 



f(X(s),u(N(s),Z(s),r(s))) 



ds 



+ 



□ 



r as r(X(s-),u d (N(s),X(s-)))dp*(s) 
< h(x), 

and taking the limit as t — > oo we obtain that h(x) > J£{x). 
Proposition 7.4 We have that v{x) = Wv(x). 

Proof: Let us show first that v(x) < Wv(x). By using the definition of VV we have for any 
T G V r (x) that 

v m+ i(x) < L a f(x, T) + H a r(x, T) + G a v m (x, T). 
Taking the limit as m f co and from the monotone convergence theorem we get that 
v(x) = lim v m+ i(x) < L a f(x,T) + H a r(x,T) + lim G a v m (x,T) 

m^oo m— >oo 

= L a f(x, T) + H a r{x, T) + G a v(x, T) 

showing that v(x) < Wt>(x). From Proposition 17.11 there exists for any m G N, G 5y such that 

Wv m (x) = L a f(x,uJ(x)) + H a r(x,uJ(x)) + G a v m (x,uJ(x)). (33) 

Fix x G -E 1 . Since u l V'(x) G V(x) C V r (x) and V r (x) is compact we can find a further subsequence, 
still written as u™{x) for notational simplicity, such that u™(x) — > G V r (x). From Proposition 

E3 

u(x) = lim v m+1 (x) 

m^oo 

= lim (L a /(x,u^(x)) + # Q r(x,u^(x)) + G a v m (x,u^(x))\ 
> L a /(x, 0) + # a r(x, 0) + G a v{x, 0) > K a (0, g)(x) = Wv{x) 



giving the result. 



(34) 
□ 



Finally we have the following theorem characterizing the optimality equation for the discounted 
optimal control problem © and showing the convergence of the truncated problems. 
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Theorem 7.5 We have that v n | J% and J%(x) = WJ%{x). 

Proof: All we need to show is that J%{x) < v(x). But this is immediate from Propositions 17.41 
andOl □ 

8 The vanishing approach 

In general it is hard to obtain a solution for the discrete-time optimality equation (see equation 
(|14p ). A common approach is to deal with an optimality inequality of the kind h > T(p,h). We 
present sufficient conditions for the existence of a solution for this inequality, using the so-called 
vanishing discount approach (see Theorem [83]). Combining this result with the connection between 
the integro-differential equation and the discrete-time equation we obtain our final main result that 
shows the existence of an ordinary optimal feedback control for the long run average cost (see 
Theorem l8.5|) . First we have the following result, which traces a parallel with the Abelian Theorem 
(see 0). 

Proposition 8.1 We have that lim Q ,;o aj^(x) < Sa( x )- 



Proof: See Theorem 1, chapter 5 in 24|. □ 



We shall add the following assumptions for the discounted problems: 

Assumption 8.2 There exists a state xq G E, numbers [3 > 0, C > 0, > ; and a nonnegative 
function b(.) such that for all x G E and a G (0,(3], p a < C, where p a = aj'^(xo) and —Kh < 
h a (x) < b(x) where h a (x) = J%{x) — J$(xq). 



We have the following propositions: 

Proposition 8.3 There exists a decreasing sequence of positive numbers [ such that p ak — ► p 
and for all x G E, lim^oo akJ ak (x) = p. 

Proof: See Lemma in [3], page 88. □ 
Proposition 8.4 Set h = lim h ak . Then for all x G E, h(x) > —Kh and h(x) > T(p, h)(x). 

fc^oo 

Proof: From Proposition 17.11 and Theorem 17.51 we have that the following equation is satisfied for 
each a > and x G E: 

h a (x) = T a (p a ,h a )(x) 

= -p a C a (x,u^(x)) + L a f(x,u%(x)) + H a r(x,u^(x)) + G Q h a (x,u^(x)), (35) 

for G 5y. 

For x G E fixed and for all k G N, u^ k (x) G V(x) C V r (x) and since V r (x) is compact we can find a 

further subsequence, still written as vQ*(x) for notational simplicity, such that u^ k (x) — > G G Y r (x). 
Combining Proposition 15.71 and equation (f35|) . 

h[x) = lim h ak (x) 

k^oo 

= lim {-p afc £ Qfe (x,u° fc (x)) + L a J(x,u^ k (x)) + H ak r(x,u* h (x)) + G ak h ak (x,v% k (x))\ 
> -pC(x, 6) + Lf(x, 9) + Hr(x, 9) + Gh(x, 9). (36) 
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Therefore, from Theorem 16.41 it follows that 

h{x) > K{p, h)(x) = T(p, h)(x) 

showing the result. 



□ 



Our final result establishes the existence of an optimal control strategy for the long run average 
cost problem. 

Let h and p be as in Propositions 18.31 and 18.41 and w = T(p, h). 

Theorem 8.5 UA,(w,h) as defined in D.3) of Definition ^. A is such that 

P = Ja(%) = A(U^{w, h),x). 

Proof: Combining Theorem 16.41 and Proposition 18.41 we obtain that the hypotheses of Proposition 
31 are satisfied for a = 0. Consequently, setting for simplicity U = Uj,(w,h), it follows that 



(x,k) 



tAT„ 



f(X(s),u(N(s),Z(s),r(s))) 



ds + 



tAT„ 



r(X(s-),u d (N(s),X(s-)))dp*(s) 



+ [ W W A T m ))] < E? X:k) [p[t A T m }] + w(x). 



Combining Proposition 18.41 and assumption 15.51 we obtain that w(x) > —pK^ — K^. Moreover, we 
have, from assumption 12.41 that T m — > oo P u a.s. . Consequently, 



' r f{X(s),u(N(s), Z(s),r(s)))} ds + / r(X(s-),u 9 (N(s), X(s-)))dp*(s) 

J J o 

< pt + pK e + K h + w(x), 



showing that p > A(U,x). From Proposition 18.11 and Proposition 18.31 we have p < Ja{x) complet- 
ing the proof. □ 



9 Appendix 

In this appendix we present several technical results required throughout the paper. 
9.1 Proofs of the results of section [3] 



We start with the proof of Proposition 13.31 

Proof of Proposition I3.3t Let {xnlneN be a convergent sequence in E with lim x n = x, 

n— >oo 

p E V r (x) and (7^ n ) neN be a sequence in V T (x n ). Define the sequence (p n ) neN in V r by 



»n{t,F) = 



1 [ -ji n (t, F) + (i - -)p(t, f n u(4>(x n , t))) 

n n 



a(n, t) 



where (t,F) € R + x B(U), and a(n,t) = ± + (1 - U(0(x„, t))). 

Clearly, we have that p n € V r {x n ). Moreover, by using the hypothesis it is easy to check that 



lim 

n— >oo 



R+ JV 



g(t, u)p n (t, du)dt 



g(t, u)p(t, du)dt, 
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for all g G L 1 (M + ; C(U)). Therefore, from Proposition D.2 in Iff] the multifunction x G E — > 
V r (x) C V r is lower semicontinuous. From Corollary III. 3 in 0], this multifunction is measurable 
and so from Proposition D.4 in [3] : /z G V r (x),x € £7} € <B(-E) x S(V r ). Finally, by using 

assumption E2] it can be shown easily that /C = { (x, 0) : G V r (x),x G E 1 } G B(-B) x B(Y r ), 
showing the result. □ 

We present next the proof of Proposition 13.41 First we need the following lemma. 

Lemma 9.1 Let h G M(R + x E x V x R+ x U)+. Then the mapping 

(t, x, n) G R+ x E x V r — > / / h(t,x, n,s,u)fi(s,du)ds eR + 

Jr + Jv 

is B(R + x E x V) -measurable. 

Proof: Let W be the class of functions /i G B(R + xExFx R + x U) such that the mapping 
{t,x,n) G R+ x E x V r -> / e~ s / h(t,x,fj,,s,u)n(s,du)ds G R 



is i3(R+ x E x V r )-measurable. This set is closed relative to addition, multiplication by constants 
and bounded pointwise passage to the limit. Consider the class C of functions h such that h = h\h2 



with hi = B(R+ x E x V r ) and /i 2 = B(R+ x U). Then from Lemma A.3 in [13j, C C H. 
Moreover, C is closed relative to multiplication. Applying Theorem T20 in [2l|, it follows that 
H contains B(R+ x E x V r x R + x U). Consider g G M(R+ x £ x V r x R+ x U)+ and define 
gk(t, x, /x, s, w) = (e s g(t, x, fj,, s, u)) A k. Therefore the mapping 

(t, x, (j,) G R+ x E x V r — > / e~ s / gk(t,x,fi,s,u)fj,(s,du)ds£R + 

Jr+ Jv 

is £>(R+ x E x V r )-measurable. By using the monotone convergence theorem, it follows 



lim / e s / gk(t,x, fi,s,u)fj,(s,du)dt = / / g(t,x, fi,s,u)[i(s,du)ds, 
k ^°° Jr + Jv Jr + Jv 

showing the result. □ 

Proof of Proposition l3T4t Applying Lemma 19.11 to h(t,x, fj,,s,u) = I{ s < t y\((j)(x,s),u) implies 
that the mapping A^(x,t) defined on R + x E x V r with value in R is measurable with respect to 
i?(R+ x E x V r ). There is no loss of generality in assuming that g is positive. Clearly, Qg(-, ■) is 
measurable with respect to B(E x U). 

Therefore, for the function h(t, x, fj,, s, u) = I^ s<tt ^-je~ as ~ A ^^ x,s ' ) X((p(x, s),u)Qg{4>{x, s),u), Lemma 
19.11 shows that the mapping 

(x, n) G E x V -> / ' e- as - K ^ x ^\Qg(4){x, s),n(s))ds G R+ 
J o 

is B{E x V r )-measurable. Moreover, the mapping 

{x,n,u) G E x V x U -» e- at *^- A ^ x ' u ^Qg(^(x,U(x)),u) G R+ 

is clearly B{E x V r x U)-measurable. Consequently, it follows that G a g(x, 6) defined on E x Y r 
with value in R is B(E x Y r ) -measurable. By using the same arguments, the same property can be 
shown for the mappings L a wi(x, 0), and H a W2(x, 0). □ 
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9.2 Proofs of the results of section [31 

The next lemma applies the semi-group property of the flow 4> (see equation ([I])) into the operators 
operators G a , L a , and H a (defined in equations ©, (fTUl) . and ([TT]) ). Recall also the definition of 
[@}t in ©. 

Lemma 9.2 For any a > 0, x £ E, t € [0, i*(a?)), 6 = (fi,Hd) G V r (x), and 5 G M(E) bounded 
from below, we have that 



C a (x,@) = / e- as - Ati( - x ' s Us + e- at - A>1 ^C a ((f)(x,t), [&) ), (37) 

JO 

L a /(x,e) = r e - QS - AM ^ s )/^(x, s ), / i( s ))d s + e- Qt ~ AM ^L a /(<A(x,t), [0] ), 

JO 

tf Q r(x,6) = e- at - AM ^F a r(0(x,t),[9] t ), 

G a g(x,e) = [ e - QS - AM ^ s )AQ 5 (0(x, S ),^( S ))d S + e- at - AM ^G a5 (( ( / ) (x,t)),[e] t ). 



Proof: For any i££, and = € V r (x), by using the semi-group property of <fr, we have 

for i + s < t*(x) 

A^(x,t + s) = / A(0(x, 0), + / + X((t>(x,e),n(£))di 



= A^{x,t)+ / \(cf>{(f>(x,t),6),{i(e + t))d£. 
Jo 

Remark that t*(x) — t = t*(<fi(x,t)). Consequently, combining the previous equation and Definition 
13.11 we obtain for t E [0, U(x)) 

r t r U{<j>{x,t)) 
C a (x,e) = / e~ as ~ A ^ds + / e -*(t+s)-A»( x> t+s) d ^ 



JO 

t r UU(x,t)) 
e -as-A»(x,s) ds + e -at-A^(x,t) / g-as-Jo H<t>(<t>(x,t),t),v,{e+t))d£ 

Jo 

t 

e- as - A ^ x ' s) ds + e- at - A ^ x ^C a ((l)(x, t), [e] t ). 
showing equation (|37|) . The other equalities can be obtained by using similar arguments. □ 



We present next the proof of Proposition 14.21 

Proof of Proposition 14. 2t From Lemma 19.21 it follows that for any x € E, t € [0, £*(x)), and 
T=(u,u d ) 6V(x) 



p£ a (x, T) + L a /(x, T) + tf a r(x, T) + G Q /i(x, T) 

p£ a {<Kx,t), [T] t ) + L a f(<f>{x,t), [T] t ) + H a r{4>(x,t), [T] t ) + G a h{<j>{x,t), [T] t 



-at-A"(x,t) 



+ 



t 

-as- A"(a;,s) 



-/> + /(0(»> «), K s )) + A(0(x, s), v(s))Qh(4>{x, s),v(s)) 



ds. (38) 



Notice now that we must have 

= - P C a ((j>(x,t), [f (x)] 4 ) + L a f(cf>(x,t), [t(x)] t ) + H a r(cP(x,t), [f (x)] t ) 

+G Q / l (0(x,t),[f(x)] t ) (39) 
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otherwise that would contradict the fact that the infimum is reached in equation (|17[) for f (x). 
Consequently, by taking T = T(x) in equation (|38|) we obtain equation (Tl9l) . 

From assumption 12, 31 we have that for all x G E and t € [0,i*(x)), e -A ^ (a:) > and so equation 
(fT9l) implies that for all x & E, w(4>(x,t)) is absolutely continuous on [0, i*(x)). Since there exists 
a constant G M+ such that — iT/j < /i, it is easy to obtain 



/ 

•70 



Consequently, if i*(x) < oo, by using assumption 12.31 the limit of w((j)(x,t)) as t — > i*(x) exists in 
R, showing that w G M ac (E). 

Let f G V(x). Define v by = L 0>t \(s)i/(s) + Ir tj00 r(s)7(x, s). Then P G V(x) and T defined by 
(#(a:), 7a(x)) belongs to V(x) and satisfies [T] = [r],. Consequently, combining (fTTl) . (f38l) and 
dMD, it follows that for all v G V(x) 

w(x) < —pC a {x, T) + L a f(x, T) + H a r(x, T) + G a h(x, T) 



-p + /(0(^) s ))K s )) 



+ X((j)(x, s),v(s))Qh((f)(x, s), u(s)) 



ds. 



Now from the previous equation and (|19p . we obtain equation ([20 



□ 

We present next the proof of Proposition 14.31 

Proof of Proposition 14. 3t Set w = T a (p,h). By hypothesis, w is bounded from below, and 
so jU(t,x,k) is well defined. For U = {u,uq) G U, defined for y = (x,z,s,n) G E, /^(y) = 
f(x,u(n,z,s)), r^iy) = r(x,uo(n, z)), h(y) = h(x), w(y) = w(x), and for t G [0,U(x)] A u (y,t) = 
A u (x,n,t). Clearly, we have that j(/(t,x,k) = w(x) > h(x) for all (t,x,k) G M+ x E x N. Now 
assume that for m G N, J^(t,x,k) > h(x) for all (t,x,k) G M+ x E x N. Define x = (a;, x, 0, k), 
then 



{*>Ti} 



{t<Ti} 
* AT m _|_ i 











/^(x, a)) - p + e- Qi w}(^(x, t)) 



(is + 



- as r u (X u (s-))dp*{s) 



o 



+ e - ttiAT »+ 1 «;(F(iAr m+1 )) 



Therefore, 



Jm+i{t,x, k) — E^ x k ^ 



+ I {t>T 1 } 
+ I {t>T 1 } 



I{t<Tl} \j e ~ aS F s)) " T s + e ~ at ™(^> *)) 

/^(x, s)) -p]ds + I {Tl=u(x)} e- at *^$(x, U(x))) 

tf\T m +\ 







F(X U (s))-p 



e 

+ e-' xtAT ™+ 1 w(X u (tAT m+1 )) 



ds + 



(40) 
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However, by using the strong Markov property of the process {X u (t)}, it follows that 

"*AT m+ i 



ht>T^~ aTl & -T 1 ,X?,k + l) = E^ 



v 



fc) 



I{t>Ti} 



ds 



tAT„. 



e- as r u (X u (s-))dp*(s) + e- atAT ^w{X u (t A T m+1 )) \F* 



(41) 



Combining equations (00]) and (jUJ), and the fact that I{ t > Tl }J^(t - Ti, Xf , fc + 1) > /{ t > Tl }/i(Xf ) 
we obtain 



jg +1 (t,x,k)>EV 



u 

(x,k) 



iATi 



/^(x, a)) - pj + / {t<Tl} e- Q ^(^(x , £)) 



(42) 



However, 



t/\T\ 



f" (</>(£ , s)) -pds + I {t<Tl} e- M w{(t>(x, t)) 



tM»(x) 



Fi^x, s)) - p] e- as - 1U ^ds + I {t<u{x)} e- at ~ 1U ^w$(x, t)), (43) 



and 



E: 



^>Tae- QTi /i(xf) + / {t > Tl=t4a;)}e -^( a; )f t/ (^x,t,(x))) 

rt At* (x) 

aU{x) - KU{£Mx M^(xMx)))I { t>u{x)} + / Q u h^(x,s))X u (^(x,s))e- as - AU ^ds 



(x,k) 



+ e- <rt '^- Xir ^^Q I ^(0(x,t,(x)))I {t > t . W} . 

Combining equations (|42|) - (|44|) . it follows that for t £ 

rtAt* (x) 



(44) 







/^(z, s)) - p + Q u h{<j>% s))\ u (<j>(x, s)) 



+ / {t>t,(x)} e 



-at, (x)-A ,7 (£,t,(x)) 



Q u h((/)(x, U(x))) + f (0(x, U (a?))) 



+ ^<t, ( , ) }e- at - X[/{£ ' i) ^(^,t)) 



iAi,(x) 



-as— A"fe (x,s) 



+ ^ W }^ (X) " A " (X,i * W) 



p + /(0(x, s), i/fc(s)) + A(0(x, s), v k {s))Qh(4){x, s), v k (s)) 
Qh((f>(x,t*(x)),ug(k,x)) + r(cf)(x,t*(x)),Ud(k,x)) 



+ / {t<t,(x)} e 



-at-A u k ( x ,t) 



w((f>(x,t)), 



ds 



(45) 



with = n(A;, x, .). Clearly, )/&(.) E V(x). Now if t < t*(x), then by applying Proposition 14. 2|, it 
follows that J^ +1 (t,x,k) > w(x) > h(x). If t > t*(x), then by defining T = {v k ,UQ(k,x)) € V(x) 
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and by using equation ([2"T]) we have 

ft* (x) 

jV +1 (t,X,k)> / ' e -««-A" fc (*,«) - / , + /( ( ,(.,-..s).„ / (.s-)) + A(.,(.r..s).,, / (.s)K^(o(.;-..s).// / ,.(.s)) 







~P + f{<P{x, s), Vk{s)) + X((f>(x, s), u k (s))Qh((j)(x, s), v k {s)) 

Qh((f)(x, t*(x)),Ug(k, x)) + r((p(x, t*(x)),ug(k, x)) 
= —pC a (x, T) + L a f(x, T) + H a r(x, T) + G a h(x, T) > w(x) > h(x), 

showing the result. □ 



We present next the proof of Proposition 14.41 

Proof of Proposition 14. 4t From Lemma A. 3 in [l3| and item (ii) of Lemma 3 in [2^] it follows 
that the mapping u defined by u(n, x, t) = 7(2:, t) belongs M(N x E x M + ; U) since T = (7, jg) £ «Sy 
Clearly, u(n,x,t) G U((f>(x, t)). Moreover, ug defined by ug(n,x) = ^g{x) belongs M(N x E;U) and 
satisfies ug(n,x) £ l]((p(x, Therefore, U = (u,ug) belongs to U. 

Set w = T a (p, h). It is easy to check that w is bounded from below, and so x, k) is well defined. 
Remark that Jq (t, x, k) = w(x) < h{x) for all (t, x, k) E R+ x E x N. Now assume that for m 6 N, 
J%(t, x, k) < h(x) for all (t, x, fc) <G M+ x J?xN, then by using this hypothesis, it follows that the 
inequalities in equations (jlSj) . (l4~5|) can be inverted for the control process given by U. Consequently, 
if t < t*(x), the last statement of Proposition I4T21 implies J^ +1 (t,x,k) < w(x) < h(x). If t > t*(x), 
then 

J^ +1 (t,x,k) < -pC a (x,f(x)) + L a f(x,f(x)) + H a r(x,t(x)) + G a h(x, f (x)) = w(x) < h(x), 
showing the desired result. □ 

9.3 Proofs of the results of section [5] 

We present next the proof of Proposition 15.71 
Proof of Proposition 15. 71 

Item a) For all < t < t*(x), if t*(x) = 00, and all < t < t*(x), if t*(x) < 00, we have from 



assumptions 12.31 and 15.11 that 

lim / / X(4>(x, s), u)fi n (s, du)ds = \ \ \(<fi(x, s),u)fj,(s,du)ds, 

~~ '0 JV(4>(x,s)) JO Jv(c/>(x,s)) 



n^oo 



or, in other words, 

lim A'Wx,*) = K^{x,t). 

n—*oo 

From items a) and b) of assumption IS~5l we have e ~ an *~ AM " ^ < e~ h SOKM)^ an d 
/•*.(») t 

/ e ~ fo £(<t>( x ' s )) ds (lt < oo). Consequently, by using the dominated convergence theorem we 
Jo 

obtain 

lim c a ( x ,e n ) = / hm e -ant-A"»(x,t) £ft = / e - a *" AM (^)^ = /: Q ( 2 ;,e), 

showing item a). 
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Item b) We have from assumption 15.21 that there exists a sequence of measurable functions (/j)j€N 
such that for all y G E, fj(y, .) f f(y, .) and .) G C(l%)). We have for (n,j) G N 2 , x G E, 

Jo 







However, item a) of assumption 15.51 gives 

| e -a„t-A^M) _ e ~«*-A ,1 (^)|/ i (0( :r) t) )/in ( t )) < 2e-tit(<f>M)ds sup f^ X}t ),a). 

aeV(4>(x,t)) 

By combining item c) of assumption 15.51 and the dominated convergence theorem we obtain 
lim f UX [ e -a»*-A^(*.*) _ e - Qt - AM ( a: .*)]/ i (0(x,t), Atn (t))^ = 0. 



n— >oo 



Therefore, 



ft* (x) 



However, remark that L an f(x,Q n ) > L an fj(x,@ n ), and the result follows by using the monotone 
convergence theorem. 



Item c) Let us consider first that t*(x) = oo. From item b) of assumptio ri5.5l and remark [576 



e 



-A^(x,t*(x)) < e -J* W {W»,i))J. = o and e -AM( x ,t.(x)) < g - £(0(x, S ))cfc = Q 



and the result follows immediately since H an r(x, n ) = Hr(x, 0) = 0. 

Suppose now that t*{x) < oo and set z = <f){x,t*{x)). We have from assumption 15.31 that there 

exists a sequence of measurable functions (?j)jeN such that for all y G dE, rj(y,.) ] r(y,.) and 

rj(y,.) G C(U(y)). Consequently, r(z,fj,Q in ) > n{z,^d,n), and so lim r(z,fia,n) > r i{ z ,Hd)- From 

n—*oo 

the monotone convergence theorem we obtain c). 
Item d) First notice that 

lim L Qn X(x, e„) = LA(x, G). (46) 

n— >oo 

Indeed, notice that L an (A+a n )(x,0 n ) = i_ e -«n**(a : )-A^(x,t,(x))_ Considering first that i*(x) < oo, 
we have that 

lim L Qn (A)(x,0 n ) = lim L Qn (A + a n )(x, n ) - a£ a (x, 0) 

n^oo ra— >oo 

= L(A + a){x, 0) - a£ Q (x, 0) = LX(x, 0). 



If i*(x) = oo then 1 = L an A(x, n ) = LX(x, 0), showing 
Set h ak = h ak + K h , h = h + K h and g k = infj>fc (thus <? fc | h and g fc < ^o!„ for n> k). By 
hypothesis, ^(y) > for all y G E. We have that is the limit of a nondecreasing sequence of 
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measurable bounded functions £ M(E). Set X m (y, a) = m A A(y, a). From assumptions 15.11 and 
5.41 we have that for each k, i and m and y € E, X m Q9k,i(y, ■) is continuous on U(y). Assumption 
and the fact that for each k, i, g k { is bounded above by, say Mf-i, yields 

0< f U X e -S$M*,'))*> sup (\ m Qg k ^( x ,t),a))dt 

Jo aeV(<t>(x,t)) 

ft* i x ) 

<mM ki e-ft*M*'''» da db<oo. 



Jo 

Since (\mQgk,i)(y,a) > and is continuous in a we have from the proof of b) (replacing / by 
^ m Q9k,i) tnat Mm L an (X m Qg kji )(x,Q n ) > L(X m Qg k>i )(x, 0), and thus, recalling that <? fc)i < /j Qn 

for n > k and A m < A, 

lim LQ, n (AQ/i an )(x,9 n ) > L a (\ m Qg kji )(x,@). 

n^oo 

From the monotone convergence theorem and taking the limit over m, i, k we get that 

lim L an (XQh an )(x,e n ) > L a (XQh)(x,Q). (47) 

n^oo 

Notice now that 

L Qn (XQh an )(x,Q n ) = L an (XQh an )(x,Q n ) + K h L an (X)(x,Q n ) 

and similarly 

L a (XQh a )(x, 9) = L a (XQh a )(x, 9) + K h L a (X)(x, 9). 

By combining (|46|) and (|4T|) we get that lim ^^^, L n ^ (XQh an )(x, 8 n ) > L a (XQh)(x, 0). Using 
similar arguments as above and c) we can show that 

hm H an h an (x,&n) > Hh(x,Q). 
completing the proof of d). □ 

9.4 Proofs of the results of section [6] 

We present first the proof of Proposition 16.31 



Proof of Proposition [tOl From assumption 13.21 and Proposition 13.41 it follows that the mapping 
V defined on /C by 

V(x, 9) = - P C a (x, 9) + L a f(x, 9) + H a r(x, 9) + G a h(x, 9) 

is measurable. Moreover, by using Corollary 15.81 it follows that for all x 6 E, V(x, .) is lower 
semicontinuous on V r (x). Recalling that V r (a;) is a compact subset of V r and by using Proposition 
D.5 in la], we obtain that there exists 9 € <Sy such equation ([25]) is satisfied. 
The rest of the proof is similar to the proof of Proposition 14.21 and it is, therefore, omitted. □ 

Before presenting the proof of Theorem 16.41 we need the following auxiliary results. 
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Lemma 9.3 Assume that w G M ac (.E). Then there exists a function X(w) in M.(E) such that for 
all x G E, and t G [0, £*(#)) 

w((f>{x,t)) -w{x) = [ X{w)((t){x,s))ds. (48) 



o 



Proof: Define 



w + (x) = lim n\w((f)(x, t*(x) A )) — w(x)] 

rw+oo L n + 1 

w~(x) = lim n[u>(0(x, i*(x) A -)) — w(x)] 

n— ++oo 71+1 

Since {u;(0(:c, t*(a;) A ^j))} neN is a sequence in M(.E), then w + (x) and ii;~(x) are Borel measur- 
able functions from E into M U {oo} U {— oo}. Consequently, the set T> w = |s G E : = 

njx <E £ : G r| belongs to B(E). 

Define the function X(w){x) by 



X(w)(x) 



dw((p(x,t)) 



w ) 



^ |t>0; it X £ L>. 

g(x), otherwise, 



where g is any function in M.(E). 

Clearly belongs to M(E). Since to G M ac (.E), there exists a set T™ G B([0,U(x))) such that 

Meb{{Tx ) c H [0,t*(x))) = and w(<fi(.,x)) admits derivatives in T™. Consequently, for any x E. E, 
and to € T™, we obtain that 4>(x,to) G and 

lim I t + e)) - to))] = ^Mfe^ | = ATH^x, *,)). 

S8e>0 

Therefore, X(w) satisfies (|48p showing the result. □ 
Lemma 9.4 For any \i G V(U(x)) and x G E, A(x,/x) < oo. 

Proof: From assumption 12.11 U(x) is a compact subspace of U, and from assumption 15. 1\ X(x, .) : 
U(x) I—*- M+ is continuous. Therefore there exists a G U(x) such that max agU ( x ) X(x, a) = X(x, a) 
and thus < X(x,/J,) = Jjjm X(x , a) /J,(da) < X(x,a). □ 

Lemma 9.5 Suppose that h G M.(E) is bounded from below by Kh- Then 
inf \ f(x, a) + X(x, a)Q(h)(x, a) — X(x, a)w(x) > 

aeU(x) I ' ' J 

= inf < f(x,u) — X(x,u)w(x) + XQh(x,u)>, (49) 

fJ,€P(V(x)) I J 

inf {r(4>(x, tJx)), a) + Qh(4>(x, t*(x)),a)} 
aeU(<t>(x,t*(x))) 

= inf {r((j)(x,tjx)),a) + Qh((j)(x,U(x)),u)}. (50) 

f«e7>(U(0(x,t„(a:)))) 
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Proof: Set for simplicity, -d(x,a) = f(x,a) + X(x,a)Q(h)(x,a) — X(x,a)w(x). Notice that from 
LemmaEU for any p G V(U(x)), X(x,p) < co and thus, recalling that / and h + Kh are positive, 

f(x, p) - X(x, p)w(x) + XQh(x, p) = f(x, p) + XQ(h + K h )(x, p) - X(x, p)[w(x) + K h ) = 



(f(x,a) + X(x,a)Q(h + Kf l )(x,a) — X(x,a)(w(x)+Kj l ))n(da)= / i!)(x,a)p(da). (51) 
>V(x) ' Jv(x) 

But as in Lemma 5.7 of we have that 

inf ■&(x,a) = inf / ■d(x,a)p(da). (52) 
Combining ([5T[) and ([52]) we get ([4*9[) . Similarly we have ([50]) . □ 



We present next the proof of Theorem [67 

Proof of Theorem 16. 41 According to Proposition 16. 3[ there exists G Syr such that for all 
x G E and t G [0, t*(x)) we have 



e-^^V^i)) -«;(*) /V«-A«">(*,«) 







/)- f(cj)(x,s),fl(x,s)) 



-XQh((p(x,s), £i(x,s)) 



ds, (53) 



where 0(x) = (p(x), pq(x)). Since w G M ac (.E), it follows from Lemma 19.31 that there exists a 
function X(w) in ~M(E) satisfying equation (|48p . Therefore, we obtain from equation (j53H that 

Afu;(</>(x, t)) - [a + A((/)(x, t),p(x, t))]w(4>(x, t)) = -f((p(x, t), fi(x, t)) - XQh(cj)(x, t),p(x, t)) + p, 

r] — a.s. on [0, t*(x)), implying that 

— Xw((p(x, t)) + aw(<p(x, t)) 



> inf 

/^(u^x,*))) 



/(</)(x, t), p) - X(4>(x, t), p)w((j)(x, t)) + XQh(4>(x, t), - p. 



However, remark that 



inf | f((f>(x, t),p)- X((j)(x, t),p)w((p(x, t)) + XQh((p(x, t), p) \ - p 
M ep(u(<Xz,i))) 

= inf \f((f)(x,t),a)-X((f)(x,t),a)[w((j)(x,t))-Qh((f)(x,t),a)]\-p 

Consequently, by considering the measurable selector u G 5u given by u = u(w, h) (see Definition 
EQ Dl)), we have 

— Xw{(f)(x, t)) + aw{x) 

>~P + f{<f>(x, t), u(<t>(x, t))) - \(</>(x, t),u{4>(x, t))) [w(0(x, t)) - Qh{(j,{x, t),u(Hx, t)))] , (54) 

r] — a.s. on [0, Let H be the set in B([0, £*(#))) such that the previous inequality is strict. If 

77(E) > 0, then there would exist t G [0,i*(x)) such that 



w(x) - e- {at+A{x ^w^{x,t)) > / e- (as+A(x ' s)) [/(0(x,s),u(^(x,s))) 







+ A(<^>(x, s),u((f)(x, s)))Qh((f)(x, s),u(4>(x, s))) - p 



ds, 
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where A(cc, t) denotes / \(4>(x, s),u(4>(x, s)))ds. However, this would lead to a contradiction with 
Jo 

equation (|26|) . Thus we have 

— Xw(<p(x,t)) + aw(<f)(x,t)) = —p + f(<f>(x,t),u((/)(x,t))) 

-\(</>(x, t),u(<Kx> *))) HHx, *)) - Qh^(x, t),u(</>(x, *)))] , 
rj — a.s. on [0, t*(x)). Consequently, for all t G [0, i*(x)) it follows that 

w ( x ) = e -( ai +^))^(0(x,t)) + [* e-^ s+X( - x ' 8 ^\f{(f)(x,s),u((j)(x,s))) 

Jo L 

+ \(<fr(x, s), u(4>(x, s)))Qh(<fi(x, s), u(4>(x, s))) — p ds. (55) 

First consider the case in which t*(x) < oo. We obtain, by taking the limit as t tends to t*(x) in the 
previous equation, that the feedback measurable selector us{w, h) G 5y (see item D2) of Definition 
16, 1|) satisfies: 

w ( x ) = e -(^)+ x (*>M*)))^(z,^ 

rU{x) 

+ / e- {as+A{x ^h{(P(x,s)M^,s)))Qh(^{x,s),u(^(x,s)))ds. (56) 



Define the control Q(x) by (fi(x),p) for p G V(U(<p(x, i*(x)))) . From equation (|28|) . we obtain that 

< - p£ a (z, 9(x)) + L a f(x, 9(x)) + # a r(x, 9 (a?)) + G a h(x, G(x)), 
and by using the definition of B(x) and 0(x), it follows that 

•-A«">O».")AQfc(0(x, s ), /}(x, 8 ))<fe 



to (x) < - pC a (x, Q(x)) + L a f(x, Q(x)) + 



t,(x) 



+ e -«*.(*)-A«">(*.t.(*)) [Qfc^fo t,( x )), m) + r(0(i, t*(x)), //)] 
From equation (|27p . we have that 



(57) 



w(x) = i r 

'0 



~P + f (</>(?> s),p(x, s)) + XQh(<p(x, s),p(x, s)) 



ds 



-at-h^ x \x,t) 



Since to G M ac (£'), this yields that 



w(x) = lim / f 
t-+u{x) J 



-as-A^ ( - x ')(x,s) 



~P + f(<P(x, s),p(x, s)) + XQh((j)(x, s),p(x, s)) 



ds 



+ lim e- at - AKx) ^w{4){x,t)) 
- pC a {x, G(x)) + L a f{x, Q(x)) + 



U{x) 



- as - A * (x)( - x ' s hQh((t>(x, s),p(x, s))ds 



+ e 



(58) 



From assumption 12.31 we have that e A* 1 * OM* s )) > 0. Therefore, combining equations ()57|) . and 
([551) . it gives that for all x G £7, and p G P(U(0(x, i*(x)))) 

w;(</>(x,i*(x))) < Qh((f>(x,t*(x)), p) + r(cf)(x,t*(x)), p). 
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Clearly, by using equation (f2"5"j) , it can be claimed that the previous inequality becomes an equality 
for p = ftd(x), implying that 

w((f>(x,U(x))) = inf {r((/)(x,t*(x)),n) + Qh(</>(x,U(x)), n)} 

= inf {r(4>(x, tJx)), a) + Qh(<j)(x, tJx)), a)}. 

a&J(<f>(x,t,(x))) 

Consequently, we have that 

w((j)(x,U(x))) = r(<j)(x,t*(x)),u(</>(x,U(x)))) + Qh(<f>(x,U(x)),u(c/>(x,U(x)))). (59) 

Combining equations (j56j) and (f59l) . it follows that 

w(x) = -pC a (x,u ( j > (w,h)(x)) + L a f(x,u ( f > (w,h)(x)) + H a r(x,u</,(w,h)(x)) 
+G a h(x, u^(w, h)(x)). 



Consider now the case in which t*(x) = oo. From assumption 15,51 we obtain that the limit when t 
tends to infinity of 



-(as+A(x,s)) 



f(4>(x, s),u(4>(x, s))) + \(4>{x, s),u((p(x, s)))Qh((p(x, s),u((j>(x, s))) - p 



ds. 



exists in MU{+oo} and that w((j)(x,t)) > —pK^ — Kh for all t G [0,+oo). Therefore, by using 
equation (|55|) we obtain that 



w(x) > - e -( at+A( - x ^[pK ( + K h ]+ / e -(«+A(*.*)) 

Jo 



f((j)(x,s),u((j)(x,s))) 



+ \(4>(x, s),u((j)(x, sj))Qh((j)(x , s),u(4>(x, s))) — p 
and so, the feedback measurable selector u^(w,h) G Sy satisfies: 



ds, 



w(x) > - pC a (x,u ( f ) (w,h)(x)) + L a f(x,u ( j ) (w,h)(x)) 

+ / e- {as+k ^ s)) \{4){x,s)M<t>{x,s)))Qh{(t){x,s),u{4>{x,s)))ds 
Jo 

= - pC a (x, u^w, h)(x)) + L a f(x, u^w, h)(x)) + H a r(x, u^w, h)(x)) 
+ G a h(x,u^(w, h)(x)). 



This shows that 



W(X) 



-pC a (x, u^w, h)(x)) + L a f(x, u^uu, h)(x)) + H a r(x, u^w, h)(x)) 
+G a h(x,u ( f > (w, h)(x)). 



In conclusion, since Y(x) C Y r (x) it follows that TZ a (p, h)(x) < T a (p,h)(x). However, we have 
shown that u^(w, h) G Sy satisfies 

K a (p,h)(x) = -pC a (x,u (j) (w,h)(x)) + L a f(x,u ( j > (w,h)(x)) + H a r(x, u$(w, h)(xj) 
+G a h(x,u ( p(w,h)(x)), 



which is the desired result. 



□ 
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